Adaptive Routing Using Expert Advice

Adaptive routing algorithms are of great importance in the maintenance of packet-switched communication networks. A sufficiently flexible algorithm can yield increased quality of service (QoS), such as reduced packet loss ratio or delay, even in case of link failures or substantially changed traffic scenarios. These algorithms require constant monitoring of network state, and the measured information is combined to update the routing tables. Such combinations can be done in a number of ways, from very simple heuristics to more complicated methods, such as neural networks or reinforcement learning or sequential decision methods. In this paper we survey the latter. In sequential decision (prediction) problems in general, a decision maker has to perform a sequence of actions. After each action, the decision maker suffers some loss, depending on the response of the environment. Its goal is to minimize its cumulative loss over a sufficiently long period of time. Adaptive routing can naturally be cast as a sequential decision problem, as for each packet the routing algorithm has to choose a path from source to destination on which the packet is to be sent. The loss corresponding to the decision is the value of the service parameter we wish to minimize, such as the delay on the path or whether the packet is lost due to insufficient buffer size. In this paper we consider sequential decision problems in the adversarial setting where no probabilistic assumption is made on how the loss of the decision maker is generated, and the goal is to perform well relative to a set of experts for all possible behavior of the environment. More precisely, the aim of the decision maker is to achieve asymptotically the same average loss as the best expert. To solve this problem, the decision maker has access to the decisions of the experts before making his own, and hence can combine them. However, it is impossible to know in advance the performance of the experts. Yet, the experts’ advice can be combined such that the average loss of the combined algorithm is asymptotically not larger than that of the best expert over a sufficiently long period of time. The first theoretical results concerning sequential prediction are due to Blackwell [1] and Hannan [2], but they were re-discovered by the learning community only in the 1990’s, see, for example, Vovk [3], Littlestone and Warmuth [4] and Cesa-Bianchi et al. [5]. These results show that it is possible to construct algorithms for sequential (online) prediction that predict almost as well as the best expert. The main idea of these algorithms is the same: after observing the past performance of the experts, in each step the decision of a randomly chosen expert is followed such that experts with superior past performance are chosen with higher probability. In the routing problem, the decisions of the experts can be defined as paths from source to destination, and in the simplest case, one can define one expert for each path. In that case, competing with the best expert results in algorithms that have at least the same asymptotic performance as the best fixed path. Stronger results can be obtained by defining more flexible (‘more clever’) experts, for example by allowing the decisions of the experts to change in time. In this way we can compete with the performance of time-varying paths. In this paper we give an overview of expert algorithms and describe their applications in adaptive routing. For simplicity, we will concentrate on minimizing the average end-to-end delay between two dedicated nodes of the network, but the results can be extended to any other QoS parameters in a